TL;DR During my Masters, I built a device able to quickly and easily estimate the complex visual properties of materials. This device allows capturing the surface of an object to reuse it in a physically based rendering context. Our SVBRDF capture system could help video game and CGI artists to replace the tedious process of handcrafting virtual materials.
I've been working on my Masters in computer vision and deep learning for more than two years now. The problem we try to solve is material appearance estimation in the wild. The goal is to estimate a representation of the surface of an object, including it's reflectivity properties. I am proud to show some of the result of these last years.
To introduce our work, here is a simple use case: In video game and movie studios, artists design virtual representations of materials. These representations contain the surface normal angles, diffuse albedo, specular albedo, roughness, etc. The process can be tedious. With our custom device, La Couronne, you point at an object with an interesting material and capture images of it in ~1sec. Then, our deep learning algorithm processes the captures and outputs a representation of the material that fits with their rendering engine.
The problem we try to solve is the estimation of the parameters associated to the physically based Spatially Varying Bidirectional Reflectance Distribution Function (SVBRDF). In simple terms, we have a bunch of images of an object and try to estimate the data that represents it. This data (the SVBRDF) is similar to a texture map in a video game.
SVBRDF estimation is based on Photometric Stereo (PS). PS estimates the geometry of an object from multiple images. In a set of images, the camera and the object do not move. Only the light position varies and this produces various shadows and different reflections. In a similar way, SVBRDF estimation methods use multiple images to estimate not only the geometry but also the reflectivity properties of the object.
Previously, complex capture systems were required and processing algorithms needed a lot of images (100+) to output decent results. Our method uses a handheld capture system and only a few images which are quick easily captured.
Most approaches in the literature who also use a low number of images are trained purely on synthetic data, which, while diverse and realistic, is often not representative of the richness of the real world. Our custom capture device enables testing these methods on real materials.
Finally, we propose a new deep learning model and approach which significantly outperforms previous methods for SVBRDF estimation on real materials.
If you are interested, here is the paper homepage.
Prototyping album:
Supplementary video:
More videos of the prototypes can be found in this google photos album.
More information on the dataset here: dataset page.