Section: Application Domains

3-D video processing

Participants : Frédéric Devernay, Matthieu Volat, Sylvain Duchêne, Sergi Pujades-Rocamora.

Stereoscopic cinema has seen a surge of activity in recent years, and for the first time all of the major Hollywood studios released 3-D movies in 2009. This is happening alongside the adoption of 3-D technology for sports broadcasting, and the arrival of 3-D TVs for the home. Two previous attempts to introduce 3-D cinema in the 1950s and the 1980s failed because the contemporary technology was immature and resulted in viewer discomfort. But current technologies — such as accurately-adjustable 3-D camera rigs with onboard computers to automatically inform a camera operator of inappropriate stereoscopic shots, digital processing for post-shooting rectification of the 3-D imagery, digital projectors for accurate positioning of the two stereo projections on the cinema screen, and polarized silver screens to reduce cross-talk between the viewers left- and right-eyes — mean that the viewer experience is at a much higher level of quality than in the past. Even so, creation of stereoscopic cinema is an open, active research area, and there are many challenges from acquisition to post-production to automatic adaptation for different-sized display  [35] , [36] .

Until recently, in order to view stereoscopic 3-D video, the user had to wear special glasses. Recent advances in 3-D displays provide true 3-D viewing experience without glasses. These screens use either a micro-lenticular network or a parallax barrier placed in front of a standard LCD, plasma, or LED display, so that different viewpoints provide different images. If the characteristics of the network and the screen are carefully chosen, the user will perceive two different images from the viewpoints of the left and right eyes. Such glasses-free 3-D screens usually display between 8 and a few dozen different viewpoints.

When the 3-D scene which has to be displayed is computer-generated, it is usually not a problem to generate a few dozen viewpoints. But when a real scene has to displayed, one would have to shoot it through the same number of synchronized cameras as there are viewpoints in order to display it properly. This makes 3-D shooting of real scenes for glasses-free 3-D displays mostly unpractical. For this reason, we are developping high-quality view-interpolation techniques, so that the many different viewpoints can be generated from only a few camera positions [14] .

Our research focuses on algorithms derived from Computer Vision and Computer Graphics, applied to live-action stereoscopic 3-D content production or post-production, including  [34] :

• Live monitoring of stereoscopic video: geometric image misalignment, depth budget (i.e. limits on horizontal disparity), left-right color balance, left-right depth-of-field consistency [16] .

• Live correction of stereoscopic video: correct the above defects in real-time when it is possible, with the help of GPU-based architectures.

• Adaptation of the stereoscopic content to the display size and distance, to avoid divergence or geometric deformations  [14] .

• Novel camera setups and algorithms for unconstrained stereoscopic shooting (especially when using long focal length).

• Novel camera setups and algorithms for glasses-free 3D displays.

• Stereoscopic inpainting.

• Stereoscopic match-moving.

• Compositing stereoscopic video and matte painting without green screen.

• Relighting of stereoscopic video, especially when videos are composited.