Section: New Results
Graphics with Uncertainty and Heterogeneous Content
MultiView Intrinsic Images for Outdoors Scenes with an Application to Relighting
Participants : Sylvain Duchêne, Clement Riant, Gaurav Chaurasia, Stefan Popov, Adrien Bousseau, George Drettakis.
We introduce a method to compute intrinsic images for a multiview set of outdoor photos with cast shadows, taken under the same lighting (Fig. 8 ). We use an automatic 3D reconstruction from these photos and the sun direction as input and decompose each image into reflectance and shading layers, despite the inaccuracies and missing data of the 3D model. Our approach is based on two key ideas. First, we progressively improve the accuracy of the parameters of our image formation model by performing iterative estimation and combining 3D lighting simulation with 2D image optimization methods. Second we use the image formation model to express reflectance as a function of discrete visibility values for shadow and light, which allows us to introduce a robust visibility classifier for pairs of points in a scene. This classifier is used for shadow labelling, allowing us to compute high quality reflectance and shading layers. Our multiview intrinsic decomposition is of sufficient quality to allow relighting of the input images. We create shadowcaster geometry which preserves shadow silhouettes and using the intrinsic layers, we can perform multiview relighting with moving cast shadows. We present results on several multiview datasets, and show how it is now possible to perform imagebased rendering with changing illumination conditions.
This work was published in ACM Transactions on Graphics [2] .

This work is part of an industrial partnership with Autodesk and has been published in ACM Transactions on Graphics [2] .
A Bayesian Approach for Selective ImageBased Rendering using Superpixels
Participants : Rodrigo Ortiz Cayon, Abdelaziz Djelouah, George Drettakis.
Many recent ImageBased Rendering (IBR) algorithms have been proposed each having different strengths and weaknesses, depending on 3D reconstruction quality and scene content. Each algorithm operates with a set of hypotheses about the scene and the novel views, resulting in different quality/speed tradeoffs in different image regions. We developed a principled approach to select the algorithm with the best quality/speed tradeoff in each region. To do this, we propose a Bayesian approach, modeling the rendering quality, the rendering process and the validity of the assumptions of each algorithm. We then choose the algorithm to use with Maximum a Posteriori estimation. We demonstrate the utility of our approach on recent IBR algorithms which use oversegmentation and are based on planar reprojection and shapepreserving warps respectively. Our algorithm selects the best rendering algorithm for each superpixel in a preprocessing step; at runtime our selective IBR uses this choice to achieve significant speedup at equivalent or better quality compared to previous algorithms. The work has been published in the International Conference on 3D Vision (3DV)  2015 [15] .

Uncertainty Modeling for Principled Interactive ImageBased Rendering
Participants : Rodrigo Ortiz Cayon, George Drettakis.
Despite recent advances in IBR methods, they are limited in regions of the scene which are badly or completely unreconstructed. Such regions have varying degrees of uncertainty, which previous solutions treat with heuristic methods. Currently we attempt to develop a comprehensive model of uncertainty for interactive IBR. Regions with high uncertainty would feed an iterative multiview depth synthesis algorithm. For the rendering we will formalize an unified IBR algorithm, which provides a good quality/speed tradeoff by combining the advantages of forward warping and depthbased backprojection algorithms and includes plausible stereoscopic rendering for unreconstructed volumetric regions.
Multiview Inpainting
Participants : Theo Thonat, George Drettakis.
We are developing a new approach for removing objects in multiview image datasets. For a given target image from which we remove objects, we use ImageBased Rendering for reprojecting the other images into the target and for regions not visible in any other image we use inpainting techniques. The difficulties reside in formalizing the unified algorithm and enforcing multiview consistency. This is an ongoing project in collaboration with Adobe Research (E. Shechtman and S. Paris).
Beyond Gaussian NoiseBased Texture Synthesis
Participants : Kenneth Vanhoey, Georgios Kopanas, George Drettakis.
Texture synthesis methods based on noise functions have many nice properties: they are continuous (thus resolutionindependent), infinite (can be evaluated at any point) and compact (only functional parameters need to be stored). A good method is also nonrepetitive and aperiodic. Current techniques, like Gabor Noise, fail to produce structured content. They are limited to socalled “Gaussian textures”, characterized by secondorder statistics like mean and variance only. This is suitable for noiselike patterns (e.g., marble, wood veins, sand) but not for structured ones (e.g., brick wall, mountain rocks, woven yarn). Other techniques, like Local RandomPhase noise, leverage some structure but as a tradeoff with repetitiveness and periodicity.
In this project, we model higherorder statistics produced by noise functions. Then we define an algorithm for maximalentropy sampling of the parameters of the noise functions so as to meet prescribed statistics to reproduce. This sampling ensures both the reproduction of higherorder visual features with high probability, like edges and ridges, and nonrepetitiveness plus aperiodicity thanks to the stochastic sampling method. We are currently investigating a learning method so as to inject into the model the appropriate prescribed statistics deduced from an input exemplar image.
This ongoing work is a collaboration with Ian Jermyn from Durham University and will be submitted for publication in 2016.
Unifying Color and Texture Transfer for Predictive Appearance Manipulation
Participants : Fumio Okura, Kenneth Vanhoey, Adrien Bousseau, George Drettakis.
Recent color transfer methods use local information to learn the transformation from a Source to an Exemplar image, and then transfer this appearance change to a Target image (figure 10 (a) to (d)). These solutions achieve successful results for general mood changes, e.g., changing the appearance of an image from “sunny” to “overcast”. However, they fail to create new image content, such as leaves on a bare tree (figure 10 (d)). Texture transfer, on the other hand, can synthesize such content but tends to destroy image structure (figure 10 (e)). We propose the first algorithm that unifies color and texture transfer, outperforming both by automatically leveraging their respective strengths (figure 10 (f)). A key novelty in our approach resides in teasing apart appearance changes that can be modeled simply as changes in color versus those that require new image content to be generated. Our method starts with an analysis phase which evaluates the success of color transfer on the Source/Exemplar scene. To do so, color transfer parameters are learned on this pair, and applied on the Source. The color transferred Source image is then evaluated against the Exemplar which serves as a ground truth, using texture distance metrics (textons in our case). This provides information on the localization of success and failure of color transfer on this scene. This analysis then drives the synthesis: a selective, iterative texture transfer algorithm that simultaneously predicts the success of color transfer on the Target and synthesizes new content using texture transfer where needed. Synthesis exploits a dense pixel matching between the Source/Exemplar scene, on which information is learned, and the Target/Output scene, on which we want to synthesize. The algorithm iterates between synthesizing the new scene by locally using either color or texture transfer, and improving the dense matching on the scene being synthesized. As a result, it leverages the best of both techniques on a variety of scenes by transferring large temporal changes between photographs, such as change of season and flooding. We demonstrate this with seasonal changes on vegetation (e.g., trees) and snow, and on examples involving flooding.
This work is a collaboration with Alexei Efros from UC Berkeley in the context of the associate team CRISP. It has been published in Computer Graphics Forum [9] and was accepted and presented at the Eurographics symposium on Rendering.

Simplification of Triangle Meshes with Digitized Radiance
Participant : Kenneth Vanhoey.
Very accurate viewdependent surface color of virtual objects can be represented by outgoing radiance of the surface. This data forms a surface light field, which is inherently 4dimensional, as the color is varying both spatially and directionally. Acquiring this data and reconstructing a surface light field of a realworld object can result in very large datasets, which are very realistic, but tedious to store and render. In this project, we consider the processing of outgoing radiance stored as a vertex attribute of triangle meshes, and especially propose a principled simplification technique. We show that when reducing the global memory footprint of such acquired objects, smartly reducing the spatial resolution, as opposed to the directional resolution, is an effective strategy for overall appearance preservation. To define such simplification, we define a new metric to guide an iterative edge collapse algorithm. Its purpose is to measure the visual damage introduced when operating a local simplification. Therefore, we first derive mathematical tools to calculate with radiance functions on the surface: interpolation, gradient computation and distance measurements. Then we derive a metric using these tools. We particularly ensure that the mathematical interpolation used in the metric is coherent with the nonlinear interpolation we use for rendering, which makes the math coherent with the rendered object. As a result we show that both synthetic and acquired objects benefit from our radianceaware simplification process: at equal memory footprint, visual quality is improved compared to state of the art alternatives.
This work is a collaboration with the ICube laboratory, Strasbourg, France. It was published in the Computer Graphics Journal [11] and was accepted and presented at Computer Graphics International 2015 in Strasbourg, France.
Video based rendering for vehicles
Participants : Abdelaziz Djelouah, Georgios Koulieris, George Drettakis.
The main objective of image based rendering methods is to provide high quality freeview point navigation in 3D scenes using only a limited set of pictures. Despite the good visual quality achieved by most recent methods, the results still look unrealistic because of the static nature of the rendered scenes. This project is in the general context of enriching image based rendering experience by adding dynamic elements and we are particularly interested by adding vehicles.
Vehicles represent an important proportion of the dynamic elements in any urban scene and adding an object with such a complex appearance model has many challenges. First, contrary to classic IBR the number of viewpoints is limited because all input videos must be recorded at the same time. Also, because of this limited number of viewpoints, using classic multiview reconstruction methods does not produce good results. Instead we use 3D stock models as proxy for the cars. The first step is the registration of the 3D model with the input videos. Then, using the 3D model, the input videos are processed to extract the different visual layers (base color, reflections, transparency, etc.). Finally, the objective is to find the appropriate way to combine the 3D model and the extracted layers to provide the most realistic image from any viewpoint.
This ongoing work is a collaboration with Gabriel Brostow from University College London in the context of the CRPLAY EU project and with Alexei Efros from UC Berkeley.
FingerBased Manipulation in Immersive Spaces and the Real World
Immersive environments that approximate natural interaction with physical 3D objects are designed to increase the user's sense of presence and improve performance by allowing users to transfer existing skills and expertise from real to virtual environments. However, limitations of current Virtual Reality technologies, e.g., lowfidelity realtime physics simulations and tracking problems, make it difficult to ascertain the full potential of fingerbased 3D manipulation techniques. This project decomposes 3D object manipulation into the component movements, taking into account both physical constraints and mechanics. We fabricate five physical devices that simulate these movements in a measurable way under experimental conditions. We then implement the devices in an immersive environment and conduct an experiment to evaluate direct fingerbased against raybased object manipulation. The key contribution of this work is the careful design and creation of physical and virtual devices to study physicsbased 3D object manipulation in a rigorous manner in both real and virtual setups.
This work was presented at IEEE Symposium on 3D User Interfaces [12] , and is in collaboration with the EXSITU Inria group in Paris (T. Tsandilas, W. Mackay, L. Oehlberg).

Gaze Prediction using Machine Learning for Dynamic Stereo Manipulation
Participants : Georgios Koulieris, George Drettakis.
Comfortable, highquality 3D stereo viewing is becoming a requirement for interactive applications today. The main challenge of this project is to develop a gaze predictor in the demanding context of realtime, heavily taskoriented applications such as games. Our key observation is that player actions are highly correlated with the present state of a game, encoded by game variables. Based on this, we train a classifier to learn these correlations using an eyetracker which provides the groundtruth object being looked at. The classifer is used at runtime to predict object category – and thus gaze – during game play, based on the current state of game variables. We use this prediction to propose a dynamic disparity manipulation method, which provides rich and comfortable depth. We evaluate the quality of our gaze predictor numerically and experimentally, showing that it predicts gaze more accurately than previous approaches. A subjective rating study demonstrates that our localized disparity manipulation is preferred over previous methods.
This is a collaboration with the Technical University of Crete (K. Mania) and Cottbuss University (D. Cunningham), and will be presented at IEEE VR 2016.
Compiling High Performance Recursive Filters
Infinite impulse response (IIR) or recursive filters, are essential for image processing because they turn expensive largefootprint convolutions into operations that have a constant cost per pixel regardless of kernel size. However, their recursive nature constrains the order in which pixels can be computed, severely limiting both parallelism within a filter and memory locality across multiple filters. Prior research has developed algorithms that can compute IIR filters with image tiles. Using a divideandrecombine strategy inspired by parallel prefix sum, they expose greater parallelism and exploit producerconsumer locality in pipelines of IIR filters over multidimensional images. While the principles are simple, it is hard, given a recursive filter, to derive a corresponding tileparallel algorithm, and even harder to implement and debug it. We show that parallel and localityaware implementations of IIR filter pipelines can be obtained through program transformations , which we mechanize through a domainspecific compiler. We show that the composition of a small set of transformations suffices to cover the space of possible strategies. We also demonstrate that the tiled implementations can be automatically scheduled in hardwarespecific manners using a small set of generic heuristics. The programmer specifies the basic recursive filters, and the choice of transformation requires only a few lines of code. Our compiler then generates highperformance implementations that are an order of magnitude faster than standard GPU implementations, and outperform hand tuned tiled implementations of specialized algorithms which require orders of magnitude more programming effort – a few lines of code instead of a few thousand lines per pipeline. This work was presented the High Performance Computing conference and is a collaboration with F. Durand, J. RaganKelley and G. Chaurasia of MIT and S. Paris of Adobe [13] .
Probabilistic Connections for Bidirectional Path Tracing
Participants : Sefan Popov, George Drettakis.
Bidirectional path tracing (BDPT) with Multiple Importance Sampling is one of the most versatile unbiased rendering algorithms today. BDPT repeatedly generates subpaths from the eye and the lights, which are connected for each pixel and then discarded. Unfortunately, many such bidirectional connections turn out to have low contribution to the solution. The key observation in this projects is that we can importance sample connections to an eye subpath by considering multiple light subpaths at once and creating connections probabilistically. We do this by storing light paths, and estimating probability mass functions of the discrete set of possible connections to all light paths. This has two key advantages: we efficiently create connections with low variance by Monte Carlo sampling, and we reuse light paths across different eye paths. We also introduce a caching scheme by deriving an approximation to subpath contribution which avoids highdimensional path distance computations. Our approach builds on caching methods developed in the different context of VPLs. Our Probabilistic Connections for Bidirectional Path Tracing approach raises a major challenge, since reuse results in high variance due to correlation between paths. We analyze the problem of path correlation and derive a conservative upper bound of the variance, with computationally tractable sample weights. We present results of our method which shows significant improvement over previous unbiased global illumination methods, and evaluate our algorithmic choices.
This work was in collaboration with R. Ramamoorthi (UCSD) and F. Durand (MIT) and appeared in the Eurographics Symposium on Rendering [10] .
