Perceptually Optimized Image Rendering

V. Laparra, A. Berardino, J. Ballé, and E. P. Simoncelli

This work has been published in the Journal of the Optical Society of America A:
Valero Laparra, Alexander Berardino, Johannes Ballé, and Eero P. Simoncelli, "Perceptually optimized image rendering," J. Opt. Soc. Am. A 34, 1511-1525 (2017)
More information can be found here.

A seminal version of this work was presented at :
Valero Laparra, Johannes Ballé, Alexander Berardino, and Eero P. Simoncelli, "Perceptual image quality assessment using a normalized Laplacian pyramid," SPIE, Conf. on Human Vision and Electronic Imaging, XXI (2016)
More information can be found can be found here.

ABSTRACT

This figure illustrates main idea of the project. When we take a picture of a scene in the real world (S) we have a perception of this scene in our mind (f(S)). However when we render this picture in a screen (I) the perception that we get is different (f(I)). If the screen was able to reproduce the real scene exactly (i.e. I = S) our perception would be the same (i.e. f(S)=f(I)). However due to different restrictions actual screens are far from being able to reproduce real world scene. We are aiming to find the image that can be reproduced in the screen that produce a perception as similar as possible to the original scene, i.e we want to minimize the distance between f(S) and f(I) with the restriction that IR has to be rendered in a screen.

We examine the problem of rendering photographic images, taking into account display limitations, so as to optimize perceptual similarity between the rendered image and the original scene. We formulate this as a constrained optimization problem, using an extension of a recently developed measure of perceptual similarity, the Normalized Laplacian Pyramid Distance (NLPD), which mimics the early stage transformations of the human visual system. This formulation presents a generic framework for solving several rendering problems. When rendering a typical high luminance image on a low luminance display, we find that the optimized solution automatically boosts the contrast of low contrast features without introducing significant artifacts. We show that this method can also be used to render HDR images, producing results of comparable quality to current state-of-the art methods with no manual intervention or parameter settings. We also analyze how to display images with optimized perceptual similarity while limiting the energy consumption of a display. Moreover, we show that the framework can be used under more complex constraints, like limiting the display to a very small number of luminance levels (halftoning). Finally, we take advantage of the behavior of the method and apply it to haze removal and artificial detail enhancement.

Normalized Laplacian Pyramid (NLP)

First, the scene luminances S (in cd/m2) are transformed using an exponential function, then they are decomposed using the Laplacian Pyramid (Burt and Adelson 1983) and finally, each channel is divided by a weighted sum of local amplitudes. Note the recursive structure inherited from the Laplacian Pyramid.

Perceptual distance based on NLP

Summation model employed to compute the perceptual distance between two images ( S and I) based on the NLP transform f.

CODE (Available will be available 1st of Februaty 2017)

Zip package with code in Matlab . It contains:

The NLP distance function.

Code to reproduce the results of the paper.

Images to reproduce the results of the paper.

NLP distance function in Python (Theano)

RESULTS

Rendering HDR calibrated images.

Rendering HDR uncalibrated images.

Rendering LDR images with an image acquisition model.

Rendering with a restricted number of gray levels (Dithering).

Artificial Detail Enhancement.

Dehazing.

RESULTS: Rendering HDR calibrated images.
MArk Farichild HDR image database contains calibrated HDR images, i.e. we know the luminance in the original scene.

The below figures show the original image and the tone mapping solution for the [Paris et al. 2011] algorithm and for the optimized version using NLP.

Back to index

Normal rendering	[Paris et al. 2011]	Optimized using NLP

RESULTS: Rendering HDR uncalibrated images.
HDR image values are linear with the luminance that was at the scene. However we do not know in which range of luminances were the images taken. Here we guess the maximum and the minimum values (Smax and Smin) of the luminance in the original scene for each HDR image.

The below figures show the original image and the tone mapping solution for the [Paris et al. 2011] algorithm and for the optimized version using NLP.

Back to index

Normal rendering	[Paris et al. 2011]	Optimized using NLP

RESULTS: Rendering LDR images with an image acquisition model (on McGill database).
Figures show results when rendering images using three different methods: a simple rescaled version to fit the dynamic range of the image to the dynamic range of the screen, a solution when using the algorithm proposed by [Paris et al. 2011] (using the proposed parameters for HDR rendering), and our proposed solution where the image is rendered to be as perceptually similar as possible to the original one using NLD. Images have being optimized to be visualized in a screen with maximum luminance (I_max) of 300 cd/m^2, minimum luminance (I_min) of 5 cd/m^2 and a gamma correction with g = 2.2. Note that in this case the luminaces of the original scene are known since the images in the McGill database are calibrated.

Back to index

Normal rendering	[Paris et al. 2011]	NLP optimized

RESULTS: Rendering with limited gray levels in the display.
Figures show results when rendering images with the dithering restriction (limited gray levels): using the Floyd-Steinberg algorithm [?], and optimizing the NLP distance. Images have being optimized to be visualized in a screen with maximum luminance of 300 cd/m^2 and a gamma correction with g = 2.2

Back to index

Normal rendering	Floyd-Steinberg (2 gray levels)	NLP optimized (2 gray levels)

	Floyd-Steinberg (4 gray levels)	NLP optimized (4 gray levels)

Normal rendering	Floyd-Steinberg	NLP optimized

RESULTS: Detail enhancement.
Usually in photographic images we do not have access to the original luminance that was at the scene. We can have an approach of values that are (approximately) linearly related with the original luminance, i.e. we have values that are related with the original luminance by a linear function which we do not know the the slope and the intercept (or equivalently the maximum and the minimum luminance). Here we are going to guess these values and we are going to see the effect on an image. We are going to set the minimum value to something close to zero, and we are going to render an image using different values to the maximum original luminance L0. Moreover in this case we use a color image an we use restrictions on the maximum value that each color channel can achieve.

The below figures show the effect of guessing differently what was the maximum luminance in the original scene (L0), note how increasing the luminance in the original scene increase the details as one can expect. For comparison we show also the result obtained by Paris when choosing the parameters of the algorithm to increase the detail of the image.

Back to index

Normal rendering	[Paris et al. 2011]

NLP optimized, L0 = 10^3	NLP optimized, L0 = 10^4	NLP optimized, L0 = 10^5

RESULTS: Dehazing results
Images taken in haze condition suffer from reduction of local contrast. We are going to take advantage of the ability of the proposed rendering method to increase the details in order to reduce the haze problem. As in the detail enhancement problem we are going to guess the values of the original luminance. We are going to set the minimum value to something close to zero, and we are going to render an image using different values to the maximum original luminance L0. For comparison we show also the result obtained by [Fattal 2014]. See the web page of R. Fattal for examples of different algorithms.

Back to index