DISCS: A Benchmark for Discrete Sampling

Sampling in discrete space, with critical applications in simulation and optimization, has recently aroused considerable attention from the significant advances in gradient-based approaches that exploits modern accelerators like GPUs. However, two key challenges seriously hinder the further research of discrete sampling. First, since there is no consensus for the experimental setting, the empirical results in different research papers are often not comparable. Secondly, implementing samplers and target distributions often require nontrivial amount of effort in terms of calibration, parallelism, and evaluation. To tackle these challenges, we propose \emph{DISCS} (DISCrete Sampling), a tailored package and benchmark that supports unified and efficient implementation and evaluations for discrete sampling from three types of tasks, namely the sampling for classical graphical models, combinatorial optimization, and energy based generative models. Throughout the comprehensive evaluations in \emph{DISCS}, we also learned new insights in terms of the scalability, the design principle of proposal distributions, and lessons for the adaptive sampling design. \emph{DISCS} implements representative discrete samplers in existing research works as baselines, and offers a simple interface that researchers can conveniently design new discrete samplers and compare with baselines in a calibrated setup directly