M.S. Paper: Deep SVBRDF Estimation on Real Materials

ece ml star

My Masters Paper: Deep SVBRDF Estimation on Real Materials

TL;DR During my Masters, I built a device able to quickly and easily estimate the complex visual properties of materials. This device allows capturing the surface of an object to reuse it in a physically based rendering context. Our SVBRDF capture system could help video game and CGI artists to replace the tedious process of handcrafting virtual materials.

I've been working on my Masters in computer vision and deep learning for more than two years now. The problem we try to solve is material appearance estimation in the wild. The goal is to estimate a representation of the surface of an object, including it's reflectivity properties. I am proud to show some of the result of these last years.

To introduce our work, here is a simple use case: In video game and movie studios, artists design virtual representations of materials. These representations contain the surface normal angles, diffuse albedo, specular albedo, roughness, etc. The process can be tedious. With our custom device, La Couronne, you point at an object with an interesting material and capture images of it in ~1sec. Then, our deep learning algorithm processes the captures and outputs a representation of the material that fits with their rendering engine.

The problem we try to solve is the estimation of the parameters associated to the physically based Spatially Varying Bidirectional Reflectance Distribution Function (SVBRDF). In simple terms, we have a bunch of images of an object and try to estimate the data that represents it. This data (the SVBRDF) is similar to a texture map in a video game.

SVBRDF estimation is based on Photometric Stereo (PS). PS estimates the geometry of an object from multiple images. In a set of images, the camera and the object do not move. Only the light position varies and this produces various shadows and different reflections. In a similar way, SVBRDF estimation methods use multiple images to estimate not only the geometry but also the reflectivity properties of the object.

Photometric Stereo example. Only the surface normals are estimated.

Previously, complex capture systems were required and processing algorithms needed a lot of images (100+) to output decent results. Our method uses a handheld capture system and only a few images which are quick easily captured.

Most approaches in the literature who also use a low number of images are trained purely on synthetic data, which, while diverse and realistic, is often not representative of the richness of the real world. Our custom capture device enables testing these methods on real materials.

Finally, we propose a new deep learning model and approach which significantly outperforms previous methods for SVBRDF estimation on real materials.

If you are interested, here is the paper homepage.

Prototyping album:

Usage example on radiometric calibration target ColorChecker
Custom PCB. Secondary LED controller. PCB front
Secondary LED controller PCB back
LED connected to populated PCB and connected to driver
Primary light controller (analog mux and arduino nano) with power supply for all 12 lights
Primary and secondary light controllers
Initially a Kinect V2 was used for the first prototype
Kinect V2 capture system (prototype
Closer look at prototype LED mount. The purple squid is a soldering tool with flexible arms. Not rigid enough.
The frame of the capture system (La Couronne). I started from a 3D CAD of the Azure Kinect and built around that. CAD (front)
Birth of La Couronne. 3D printed on Creality Ender 3
3D printed body for the final version
Starting assembly of the lights. Brackets were added to achieve larger angle with object.
Six lights assembled
Assembled primary light controller with power supply
Final version compared to rough draft of the final version
La Couronne assembly completed. Azure Kinect camera inserted. Not satisfied of the messy wiring. It does the job for now.
Side view with messy wiring
The device can be handheld but we mount it on a tripod most of the time
Final version with clean wiring
Side view
Back view
Usage example
usage example
La Couronne
Example of a shiny material capture (256x256 crop)

Supplementary video:

More videos of the prototypes can be found in this google photos album.

More information on the dataset here: dataset page.