Portrait Lighting Transfer using a Mass Transport Approach

Lighting is a critical element of portrait photography. However, good lighting design typically requires complex setups and significant time and expertise. Our goal is to achieve the desired portrait lighting with a relighting technique that transfers the illumination of a portrait onto another one. The novelty in our approach to this challenging problem is our formulation of relighting as a mass transport problem. We start from standard color histogram matching that only captures the overall tone of the illumination, and show how to use the mass-transport formulation to make it dependent on facial geometry. We fit a 3D morphable face model to the portrait, and for each pixel we combine the color value with the corresponding 3D position and normal. We then solve the mass-transport problem in this augmented space to generate a color remapping that achieves localized, geometry-aware relighting. Our technique is robust to variations in facial appearance and small errors in face reconstruction. As we demonstrate, this allows our technique to handle a variety of portraits and illumination conditions, including scenarios that are challenging for previous methods.

Abstract Lighting is a critical element of portrait photography. However, good lighting design typically requires complex setups and significant time and expertise. Our goal is to achieve the desired portrait lighting with a relighting technique that transfers the illumination of a portrait onto another one. The novelty in our approach to this challenging problem is our formulation of relighting as a mass transport problem. We start from standard color histogram matching that only captures the overall tone of the illumination, and show how to use the mass-transport formulation to make it dependent on facial geometry. We fit a 3D morphable face model to the portrait, and for each pixel we combine the color value with the corresponding 3D position and normal. We then solve the mass-transport problem in this augmented space to generate a color remapping that achieves localized, geometry-aware relighting. Our technique is robust to variations in facial appearance and small errors in face reconstruction. As we demonstrate, this allows our technique to handle a variety of portraits and illumination conditions, including scenarios that are challenging for previous methods.
Paper Portrait Lighting Transfer using a Mass Transport Approach. Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, Dimitris Samaras, ACM Transactions on Graphics (TOG) (to appear).
Supplementary Material Supplemental Document (PDF, 123MB) Data (coming soon) Code (coming soon)
Acknowledgement This work started when Zhixin Shu was an intern at Adobe Research. This work was supported by a gift from Adobe, NSF IIS-1161876, the Stony Brook SensorCAT and the Partner University Fund 4DVision project.