Transforming a Non-Differentiable Rasterizer into a Differentiable One with Stochastic Gradient Estimation

We show how to transform a non-differentiable rasterizer into a differentiable one with minimal engineering efforts and no automatic differentiation. To do so, we improve on Stochastic Gradient Estimation by using a Per-Pixel Loss which leverage the fact that only a few primitives contribute to a given pixel. Estimating gradients on a per-pixel basis bounds the dimensionality of the optimization problem and makes the method scalable. To track parameters contributing to a pixel, we use ID- and UV-buffers, which are often already available or trivial to obtain. With these minor modifications, we obtain an in-engine optimizer for 3D assets with millions of geometry and texture parameters.