An introduction to color spaces

LWN.net needs you!

Without subscribers, LWN would simply not exist. Please consider signing up for a subscription and helping to keep LWN publishing.

His slides started with the v4l2_pix_format structure, which describes the pixel format of the data returned by a video capture device (such as a webcam). The colorspace member of that structure, in particular, identifies the color space in which the pixel data is expressed. Developers of applications (and drivers) for media devices must specify an appropriate color space, but few developers, Verkuil said, understand what that field really means.

Color, he said, can be thought of as a signal consisting of light at one or more specific frequencies and powers. The human eye, though, does not detect all of those frequencies directly. Instead, it has three types of "cones" with sensitivities centered around three specific frequencies — nominally red, green, and blue. Light at a specific frequency and power will generate a certain level of signal from each type of cone; the three-value tuple that results is how color is signaled to the brain. As it happens, there is an infinite set of frequency/power distributions that can result in the same three values. Reproducing a color, as far as the brain is concerned, is just a matter of reproducing a specific color tuple. Photographs and video displays take full advantage of that fact; the colors they produce are not the original colors, but they are able to fool the eye into seeing the the original colors.

Color spaces

When dealing with visual data, we need a way to uniquely identify colors; that is where a color space comes in. Back in the 1920s, the CIE (Commission Internationale de L'Éclairage) performed a set of studies mapping wavelengths onto the RGB values that replicate them. Those values became the CIE RGB color space. A simple linear transformation turns CIE RGB into the CIE XYZ color space, which has a couple of practical advantages: it allows all colors to be represented using positive values, and the Y value describes the overall brightness (luminance) of the color. Among other things, it turns the range of possible colors (at a given luminance) into a two-dimensional quantity. All other color spaces are based on CIE XYZ — which was developed in the 1920s from measurements on a pool of 17 people.

In general, a color space defines three "primaries" that can be thought of as the red, green, and blue colors, though they don't always correspond to those colors. Each color space also has a "white point" describing the maximum output value for each primary. Once upon a time, color spaces corresponded to the physical properties of the phosphors found in CRT screens, but that is no longer the case.

Color spaces as described thus far are linear, with values corresponding directly to the light levels of the primaries. The human eye does not respond to light linearly, though; a doubling of the light level does not look twice as bright. If (say) eight bits are used to represent a primary value in a color space, many of the 256 available values will be wasted on tiny differences between the brightest values, while the resolution is too coarse at the dim end of the scale. So colors are often represented in nonlinear color spaces that better match how the eye responds.

If a linear color space has RGB values, then a nonlinear equivalent can be obtained by applying a "transfer function" yielding a new set of values, called R'G'B'. The primes should be used for nonlinear color spaces, but almost everybody leaves them out, with the result that nobody ever knows which kind of color space is being talked about. The transfer function is often called a "gamma function", Verkuil said, but that is not quite correct. The screen will typically apply an inverse transfer function to color values to get the actual intensities to display; needless to say, the transfer function and its inverse need to match or colors will not be displayed correctly. OpenGL programmers need to be aware that textures use linear RGB values by default, not nonlinear R'G'B'.

Video applications often deal with colors in the Y'CbCr (or YUV) "color space", but it is not actually a separate color space. Y'CbCr is derived directly from R'G'B' via a matrix multiplication; it is simply a different representation for the same color space. Even so, it seems that a color space can define its own matrix (or even more than one) for this transformation.

Verkuil went quickly through some of the more prominent standards in this area.

• The Rec. 709 color space is for high-definition television; it is, he said, "nicely done."

• The best-known color space, perhaps, is sRGB, which is typically used for computer graphics. It has the same chromaticities (primaries) as Rec. 709, but the transfer function is different.

• SMPTE 170M is the color space for standard-definition TV; it has the same transfer function as Rec. 709, but the chromaticities are different.

• BT.2020 is for ultra-high-definition television with at least ten bits for each color component. There are two separate Y'CbCr encodings defined for this color space.

Once the color space and encoding have been figured out, there is one more complication in the form of limited-range encoding. Normally, eight-bit R'G'B' color values use the full 0..255 range, but colors in the Y'CbCr encoding are compressed to fit in the narrower 16..235 range. Limited-range R'G'B' does exist in the wild, though, as does full-range Y'CbCr. The limited-range encoding is a holdover from the old analog television days, when the margin at either end was needed to handle errors. Everything is digital now, but we are still stuck with limited range encoding in a number of situations. Some transports use the out-of-range values as sentinel values for in-band signaling.

After spending a lot of time gaining a better understanding of color spaces, Verkuil added some additional fields to the v4l2_pix_format structure:

    __u32 ycbcr_enc;	/* enum v4l2_ycbcr_encoding */
    __u32 quantization;	/* enum v4l2_quantization */
    __u32 xfer_func;	/* enum v4l2_xfer_func */

These fields describe which Y'CbCr encoding is in use (if any), whether full-range or limited-range quantization is in use, and which transfer function has been applied. Now it is possible for user space to learn everything it needs to handle color spaces correctly — but user-space developers still ignore it all, he said. There is one exception to that, actually: the GStreamer developers have worked hard to get their color-space handling right.

When things go wrong

What happens if you don't put in that effort and don't get things right? There are a number of problems that can result, and it's not all the developers' fault. The names for the color spaces are confusing (CIE XYZ is not the same as CIE xyz or CIE Yxy), conversion matrices can be buggy, and, of course, there is the full range of exciting surprises that originate in hardware implementations.

One thing that often goes wrong is a confusion between SMPTE 170M and Rec. 709. These two color spaces have different primaries, leading to slight color differences. Those differences are indeed slight, though, to the point that only an expert is likely to notice them; most developers can safely ignore this particular difference. It is slightly visible on LCD screens, but completely disappears when projectors are in use.

Things go a bit further amiss when the Rec. 709 and sRGB transfer functions are confused. The differences here are more noticeable, especially toward the black end of the scale. Using the wrong Y'CbCr encoding is quite a bit more obvious; that's something that customers will notice. Using limited-range quantization when full-range is expected (or the reverse) is also quite evident. This one tends to manifest itself when somebody is displaying an Excel spreadsheet; the slight color difference between adjacent rows will vanish if a full-range signal is interpreted as limited-range.

The slides from the presentation give examples of the visible differences resulting from the above problems. The media subsystem documentation has information for developers wanting to learn more about using color spaces with video acquisition devices.

[Your editor would like to thank Kernel Recipes for supporting his travel to the event.]