Computational Photography: Basics

Quentin Bammey

From scene to rendered image

Today’s class

  • Overview / Recall of a few basic concepts:
    • Image Formation
    • Light / Illumination
    • Reflectances
    • Physical Image Sensors

Image Formation Model

Image Formation Model

  • Illuminant SPD \(E(\lambda)\)
  • Surface Reflectance factor \(S(\lambda)\)
  • Sensor sensitivity \(R(\lambda)\)

Colour signal: \[ C(\lambda) = E(\lambda) \cdot S(\lambda) \]

Sensor response: \[\rho = \int_\lambda C(\lambda)\cdot R(\lambda)~\mathrm d\!\lambda = \int_\lambda E(\lambda) \cdot S(\lambda)\cdot R(\lambda)~\mathrm d\!\lambda\]

How to obtain colour information?

Colour Image Formation Model

  • Human Visual System: psychovisual measurements \[\left(\begin{matrix}X\\Y\\Z\end{matrix}\right) = \int_\lambda C(\lambda)\cdot \left(\begin{matrix}R_X(\lambda)\\R_Y(\lambda)\\R_Z(\lambda)\end{matrix}\right)~\mathrm d\!\lambda\]
  • Camera measurements \[\left(\begin{matrix}R\\G\\B\end{matrix}\right) = \int_\lambda C(\lambda)\cdot \left(\begin{matrix}R_R(\lambda)\\R_G(\lambda)\\R_B(\lambda)\end{matrix}\right)~\mathrm d\!\lambda\]

Image Formation Model: Human vs Camera

Light (Illuminant)

Electro-magnetic Spectrum

Quantum Theory

Electrons jump between orbits. When they go from a higher energy orbit to a lower energy orbit, they emit a photon

Emitted radiation energy

The emitted radiation has energy: \[Q = h \nu = \frac{h c}{\lambda}\]

  • \(Q\): Photon energy (Joules or eV)
  • \(h = 6.623 \cdot 10^{-34} Js\) is the Planck’s constant
  • \(c = 2.998 \cdot 10^8 m s^{-1}\): speed of light
  • \(\lambda\): wavelength (in meters)
  • \(\nu\): frequency

Number of Photons (quantas)

  • The radiant power (flux) \(\Phi\) (Watts) is the energy emitted, transferred or received per time. \[\Phi = \frac{\mathrm d Q}{\mathrm d t}\]
  • Over a spectral region \(\mathrm d\lambda\), the radiant power is \[\Phi_\lambda\mathrm d\lambda = -\Phi-\nu\mathrm d\nu\]
  • As \(\lambda = \frac c \nu\), \(\mathrm d\lambda = -c\nu^2 \mathrm d\nu\), thus \[\Phi_\nu = \Phi_\lambda \frac c {\nu^2} = \frac 1 c \Phi_\lambda \lambda^2\]
  • The number of photon emitted in time per frequency interval is \[N_\nu = \frac {\Phi_\nu}{h\nu} = \frac 1 {h c^2}\Phi_\lambda \lambda^3\]

Number of Photons (quantas)

Radiometric measures

Depend on power

  • Radiant Exitance \[M = \frac {\mathrm d\Phi}{\mathrm d A_{source}} (W m^{-2})\]
  • Irradiance \[E = \frac {\mathrm d\Phi}{\mathrm d A_{receiver}} (W m^{-2})\]
  • Radiant Intensity \[I = \frac{\mathrm d\Phi}{\mathrm d\omega} [W sr^{-1}]\]
  • Radiance \[L = \frac {\mathrm d\Phi}{\mathrm d\omega\mathrm d A_{surface}\cos\varphi} [W m^{-2}sr{ -1}]\]

\(\varphi\) angle to surface normal

In addition to radiometric measures, we use spectral radiometric measures by defining the previous measures per unit wavelength or frequency interval

A reminder on solid angles

Black Body (Planckian Radiator)

  • Every object emits energy from its surface when heated to a temperature greater than 0 Kelvin
  • a Black Body is an ideal solid object that absorbs and emits EM radiations throughout the EM spectrum, so that:
    • All incident radiation is completely absorbed
    • Emission is possible in all wavelengths and directions

Black Body (Planckian Radiator)

  • The SPD are continuous functions of wavelength
  • The exact radiation emitted depends on its absolute temperature

The Three Laws of Black Bodies

  • The spectral radiant exitance of a black body is given by \[M_\lambda (T) = \frac \alpha {\lambda^{5} e^{\frac{\beta}{T\lambda}}} [W m^{-3}\]
  • \(\alpha = 3.742 \cdot 10^{-16} Wm^2\)
  • \(\beta = 0.004388 mK\)

The wavelength of the maximum emitted radiation is inversely proportional to the temperature \[\lambda_{max} = \frac {2.897\cdot 10^{-3}} T\]

As the temperature increases, the amount of radiation at each wavelength increases, with the total radiant excitance being proportional to \(T^4\)

\[M = \int_0^\infty M_\lambda(T)\mathrm d\lambda = \sigma T^4\]

  • \(M [W m^{-2}]\): total radiant exitance
  • \(\sigma = 5.67 \cdot 10^{-8} Wm^{-2} K^{-4}\): Stefan-Boltzmann constant

The Three Laws of Black Bodies: Summary

  • The spectral radiant exitance distribution of a black body depends only on its temperature
  • The peak wavelength of the emitted radiation is inversely proportional to the temperature: hotter black bodies have shorter-wavelength peaks (blue side of the spectrum), colder bodies have longer wavelength peak (red side of the spectrum)
  • The amount of radiation at each wavelength monotonically increases with the temperature
  • The total radiation is proportional to \(T^4\)

Spectral radiant Excitance

Why does this matter?

Why does this matter?

The human vision system has a Colour Constancy property: we quickly adapt to the color of the light.

In camera processing: we use White Balancing to estimate the illuminant and correct for it to mimic human colour constancy

Why does this matter?

The dress can be seen as blue and black under yellow illumination, or as white and gold under a blue illumination

By Figure design by Kasuga~jawiki; vectorization by Editor at Large; “The dress” modification by Jahobr, CC BY-SA 3.0, Link

The dress was illuminated by store light, while the saturated outside was illuminated by daylight. The white balance of the photo failed to account for the multiple exposures, and incorrectly corrected the dress as if it has been under daylight.

Why does this matter?

Claude Monet: Cathédrale de Rouen

A series of 30 paintings of the same building, at different times of day, or the importance of the illuminant before the digital era.

When taking pictures, we want to perform white balance to correct the illuminant and obtain a plausible image. However, we do not want to apply it too strongly, as we want to preserve information about the illuminant, which is part of a picture and can set its tone/mood.

Correlated Colour Temperature (CCT)

  • The colour of light emitted by a solid object can be specified by the radiation laws.
  • We assign a colour temperature to a light source by matching its spectral power distribution to the closest black body

Correlated Colour Temperature

Colour Temperature of Common Light Sources

  • Candle: 1800K
  • Tungsten Lamp: 2800 K
  • Halogen Lamp: 3800 K
  • Direct Sunlight: 4800 K
  • Photographic Film Daylight: 5500 K
  • Electronic Camera Flash: 6000 K
  • Overcast Sky Light: ~8000 K
  • Blue Sky Light: ~10000 to 20000K
  • Monitor “white”: ~5000 to 9300 K

Correlated Colour Temperature

  • The correlated colour temperature is not enough to fully qualify a light source: the full spectral power distribution is needed
  • We usually normalize different spectral power distributions (e. g. to match at 560 nm): what matters is the distribution of light more than its absolute values.

Daylight

  • There is no single colour temperature for daylight: it varies according to season, atmosphere, time of day, geographical location, etc
  • The Commission Internationale de l’Éclairage (CIE) provide standards for a few light sources with known spectral power distributions

Time of Day

CIE Standard Illuminant A (CCT 2856 K)

Typical tungsten-filament or incandescent lighting (was used in light bulbs before LED lights became common)

CIE Standard illuminants B and C

These illuminants are early/obsolete approximations of daylight: B represents noon daylight while C represents average North sky daylight.

They are no longer used much as we now have much better estimations

CIE Standard Illuminants series D

D series represent natural daylights. D65 is considered the standard illuminant.

D50 is used in the printing industry, it corresponds horizon light and applies to many lighting situations.

D55 represents mid-morning / mid-afternoon daylight

D65 is the standard illuminant, it represents noon daylight and is used when mapping in and out of different devices (sRGB colour space)

North sky daylight

CIE Standard Illuminants series D

Beyond these four canonical standards, it is possible to create D-like representations for other temperatures: All D illuminants are a linear combination of three SPDs.

While the D numbers represent the temperature (D65 stands for 6500 K), note that the exact CCT of the four canonical illuminants differ slightly from this value (the CCT of D65 is actually 6504 K). This is due to the SPDs of D illuminants being based on Planck’s law constants: our estimate of the constants is now more precise than what was known when the standard illuminants were created.

CIE Standard Illuminant E

E is a theoretical reference for an equal-energy radiator, it has constant SPD within the visible spectrum.

CIE Illuminants series FL

Series FL represent fluorescent lamps.

Standard fluorescent lamps

Broadband fluorescent lamps with a fuller spectrum

Narrow-band lamps

CIE standard illuminant series LED

LEDs have a narrow bandwidth: for a correct “white” light, several LEDs are needed (the narrow peaks of individual LEDs is particularly easy to see in the LED-RGB1 illuminant).

Phosphor-converted blue light

Mix of phosphor-converted blue light and a red LED

RGB system (mix of three colour LEDs)

phosphor-converted violet light

CIE standard illuminant series ID

Two standards representing natural indoor light. ID50 and ID65 are equivalent to outdoor D50 and D65, filtered through glass windows. The ultraviolet contents are filtered by the glass.

Other CIE standard illuminants

There are a few other standard illuminants * FL3.1 — 3.15: more different fluorescent illuminants * HP 1 — 5: high pressure discharge lamps

Light measurement

Until now, we have measured EM radiations: this is radiometry, where we equally measure all radiations equally.

In photography, we also use photometry, by weighting the measurements according to the luminosity function of the human eye \(V_\lambda\) \(\Rightarrow\) values are only sampled in the visible spectrum

Luminous Flux (power): radiant flux weighted by the human luminosity function

\[\Phi_L = 683 \int_{380}^{750}V(\lambda)\Phi_R(\lambda)~\mathrm d\!\lambda\]

Human eye sensitivity

Our sensitivity peak is around the green values: we are more sensitive to changes in contrast around the green values.

  • Before colour correction, raw images appear green because we are actually more sensitive to green light
  • We are more sensitive to variations in shade of green, thus when reconstructing images we need more precision in the green values. This is why cameras have more green sensors than red or blue.

Terminology

Radiometric term Photometric term
Radiant Flux \(\Phi_R\) (Watts \(W\)) Luminous Flux \(\Phi_L\) (Lumens \(\mathrm{lm}\))
Radiant Exitance \(M\) (\(W\cdot m^{-2}\)) Luminous Emittance (\(\mathrm{lm}\cdot m^{-2}\))
Radiant Intensity \(I_R\) (Watts per steradian, \(W\cdot\mathrm{sr}^{-1}\)) Luminous Intensity \(I_L\) (candela \(\mathrm{cd} = \mathrm{lm}\cdot\mathrm{sr}^{-1}\))
Irradiance \(E\) (\(W\cdot m^{-2}\)) Illuminance \(E\) ($=m^{-2}
Radiance \(L\) (\(W\cdot m^{-2}\cdot\mathrm{sr}^{-1}\)) Luminance \(L\) (\(\mathrm{cd}\cdot \mathrm m^{-2}\))

Reflectances

The color of an opaque object is characterized by its surface reflectance \(S(\lambda)\in[0, 1]\).

Light reflection

Diffuse

Glossy

Specular (mirror)

Surfaces are represented by bidirectional reflectance distribution functions (BRDF) with a diffuse component (reflects light equally at all angles), a glossy component (reflects light around the reflection direction), and a specular component (like a mirror, reflects like only at the reflection direction).

Lambertian surfaces

Diffuse
  • For this class, we assume surfaces to be Lambertian
  • Lambertian surfaces only have a diffuse component, thus reflect light equally at all angles. If \(L\) is the radiance or luminance, \(E\) the irradiance or illuminance, and \(S\) the reflectance factor of the surface, \[L(\lambda) = \frac 1 \pi E(\lambda) \cdot S(\lambda)\]
  • Good approximation for most surfaces, especially rough ones, but does not apply to reflective or glossy surfaces (mirrors, flat bodies of water, polished surfaces, etc.)
  • To learn more about non-Lambertian surfaces, take an advanced graphics course such as CS 440.

Lambert’s cosine Law

The irradiance/illuminance falling on a surface is proportional to the cosine of the incident angle \(\vartheta\): \[E(\vartheta) = E(\vartheta=0)\cdot\cos\vartheta\]

The surface radiance is thus \[L_R = \frac 1 \pi \int_{380}^{750} E(\lambda)\cos\vartheta S(\lambda)~\mathrm d\!\lambda\] and the surface luminance \[L_R = \frac {683} \pi \int_{380}^{750} E(\lambda)\cos\vartheta V(\lambda) S(\lambda)~\mathrm d\!\lambda\]

(\(E\) is the spectral irradiance, \(S\) the spectral reflectance factor of the surface)

Sensors

Two kinds of sensors

  • Human sensor: the eye (next lecture)
  • Imaging sensors use a photo sensitive material: silicon on photo cells arranged in rows, columns, or areas
  • Two types of sensors
    • CCD (Charge Coupled Device): rare nowadays except in specific applications
    • CMOS (Complementary Metal Oxide Semiconductor)
  • CCD pushes information to the side and reads it by row or column: larger sensor size, slower read time
  • CMOS processes information directly on the site of each pixel: faster, but pixel size is smaller (we can only capture the light that arrives to the centre of the sensor) \(\Rightarrow\) less sensitivity in low-light situations
    • Modern CMOS sensors use small lenses to focus all incoming light to the photosensitive region, removing this issue.

Pixel well

  • Each pixel is a two-layers container:
    • P-layer: silicon with a substance lacking one peripheral electron
    • N-layer: silicon with a substance having one surplus peripheral electron
  • The photon arrives to the P-layer and creates an electron hole, which interacts with the N-layer to create a charge.

Sensor response

  • Read the pixel by counting the number of electrons in the voltage
  • Silicon reacts linearly with the photon count, up to a level of saturation
  • Full-well capacity: maximum count that can be read
  • Saturation capacity: maximum count after which the sensor no longer reacts linearly
  • Sensitivity threshold: dark current and read noise, electrons that can be read even without light (usually up to 2 electrons for good CMOS sensors)
  • Dynamic range: ratio of full-well capacity and sensitivity threshold, often expressed in bits or db (log space): \[\mathrm{dyn} = \frac{\text{full-well capacity}~[e⁻]}{\text{sensitivity threshold}~[e^-]}\]

Image Formation, revisited

Image Formation Model

Image Formation Model

  • Illuminant SPD \(E(\lambda)\)
  • Surface Reflectance factor \(S(\lambda)\)
  • Sensor sensitivity \(R(\lambda)\)

Colour signal: \[ C(\lambda) = E(\lambda) \cdot S(\lambda) \]

Sensor response of a sensor \(k\): \[\rho-k = \int_\lambda C(\lambda)\cdot R(\lambda)~\mathrm d\!\lambda = \int_\lambda E(\lambda) \cdot S(\lambda)\cdot R_k(\lambda)~\mathrm d\!\lambda\]

Colour Image Formation Model

Three sensorsm sensitive to different wavelengths: * Human Visual System: psychovisual measurements \[\left(\begin{matrix}X\\Y\\Z\end{matrix}\right) = \int_\lambda C(\lambda)\cdot \left(\begin{matrix}R_X(\lambda)\\R_Y(\lambda)\\R_Z(\lambda)\end{matrix}\right)~\mathrm d\!\lambda\] * Camera measurements \[\left(\begin{matrix}R\\G\\B\end{matrix}\right) = \int_\lambda C(\lambda)\cdot \left(\begin{matrix}R_R(\lambda)\\R_G(\lambda)\\R_B(\lambda)\end{matrix}\right)~\mathrm d\!\lambda\]

Simple Image Formation

  • For most imaging modelling, we express the spectral distributions as samples taken every 10nm between 400nm and 700nm \[\rho_k(\mathbf x) = n \sum_{\lambda=1}^{31} C(\mathbf x, \lambda) R_k(\lambda) = n \sum_{\lambda=1}^{31} S(\mathbf x, \lambda) E(\mathbf x, \lambda) R_k(\lambda)\]
  • \(n\) is a normalization factor

Algebraic Notations

We express the values as vectors of size 31:

Name Continuous notation Discrete vector notation
Colour signal \(C(\mathbf x, \lambda)\) \(\mathbf c(\mathbf x)\)
Reflectance \(S(\mathbf x, \lambda)\) \(\mathbf s(\mathbf x)\)
Illumination \(E(\mathbf x, \lambda)\) \(\mathbf e(\mathbf x)\)
Sensor \(k\) sensitivity \(R_k(\lambda)\) \(\mathbf r_k\)

The sensor response is thus \[\rho_k(\mathbf x) = \mathbf c(\mathbf x)^\intercal \mathbf r_k = \mathbf s(\mathbf x)^\intercal \mathrm{diag}\left(\mathbf e(\mathbf x)\right)\mathbf r_k\]

Where \({}^\intercal\) is the matrix transposition and \[\mathrm{diag}(\mathbf e) = \begin{pmatrix} e_0(\mathbf x) & 0 & 0 & \cdots & 0 \\ 0 & e_1(\mathbf x) & 0 & \cdots & 0 \\ 0 & 0 & e_2(\mathbf x) & \vdots & 0 \\ \vdots & \vdots & \cdots & \ddots & \vdots \\ 0 & 0 & \cdots & 0 & e_{30}(\mathbf x)\end{pmatrix}\]

Colour response vector

Any physical sensor can have a number of filters (channels) with different sensitivities \(R_k(\lambda)\).

The human vision system and most cameras are trichromatic, e. g. we have three different channels.

The total colour response at position \(\mathbf x\) of such a system is thus a vector with three entries:

\[\rho(\mathbf x) = \begin{pmatrix}\rho_1(\mathbf x)\\\rho_2(\mathbf x)\\\rho_3(\mathbf x)\end{pmatrix}\]

We usually call the three responses:

  • RGB, for sensors whose peak sensitivities are in the long wavelengths (red), medium wavelengths (green) and short wavelengths (blue) parts of the spectrum
  • XYZ for sensors corresponding to the CIE colour matching functions
  • LMS for sensors corresponding to eye cone fundamentals

Summary

Summary

  • The simple image formation model is \[\rho = \int_\lambda E(\lambda)S(\lambda)R(\lambda)~\mathrm d\!\lambda\]
  • When either the illuminant, the surface of the sensor changes, so does the formed image!
  • The human vision system corrects the information for different exposures and illuminant spectra, but we need to do it ourselves in cameras.
  • Illuminants:
    • Black bodies (Planckian radiators) are ideal models governed by three radiation laws:
      • Their spectral radiant exitance distribution is a function only of temperature
      • Their peak wavelength is inversely proportional to temperature (blue light is ``hotter’’ than red light)
      • Even though the peak wavelength ``moves left’’ when the temperature increases, a hotter black body emits more light at all points of the spectrum
    • Real illuminants are described by a correlated colour temperature (CCT), which matches them visually to the closest-looking black body, but we still need their full spectral power distribution to specify them
    • Radiometric measurements describe power, intensity, excitance and irradiance throughout the EM spectrum
    • Photometric measurements are radiometric measurements weighted by the luminosity function of the Human Vision System
  • Surface reflectances
    • The surface reflectance is a factor between 0 and 1 at each wavelength, representing which portion of the light is reflected by the surface throughout the spectrum
    • For this class, we mostly consider Lambertian surfaces: reflecting light is diffused equally at all angles

Conclusion

  • Quick review of concepts in basic photography
  • Most concepts we use are simplifications of the real world.
  • Look up online or in books for extensive definitions of the concepts
  • What we have seen today will be used in depth throughout the semester