Many of the sounds we hear in daily life are generated by contact between objects. Rigid objects are often well approximated as linear systems, such that impulse responses can be used to predict their vibrational behavior. Impulse responses carry information about material and shape. Previous research has shown that impulse responses measured from objects can be used to generate realistic impact, scraping and rolling sounds. However, it has been unclear how to efficiently synthesize impulse responses for objects of a particular material and size. Here we present an analysis-by-synthesis technique that uses a differentiable impulse response synthesis model to infer generative parameters of a measured impulse response. Then, we introduce a way of representing auditory material as distributions in the generative parameter space. Object impulse responses can be sampled from these distributions to render convincingly realistic contact sounds.