Many important datasets in physics, chemistry,and biology consist of noisy sequences of imagesof multiple moving overlapping particles.In many cases, the observed particles are indistinguishable,leading to unavoidable uncertaintyabout nearby particles’ identities. Exact Bayesianinference is intractable in this setting, and previousapproximate Bayesian methods scale poorly.Non-Bayesian approaches that output a single“best” estimate of the particle tracks (thus discardingimportant uncertainty information) aretherefore dominant in practice. Here we proposea flexible and scalable amortized approach forBayesian inference on this task. We introducea novel neural network method to approximatethe (intractable) filter-backward-sample-forwardalgorithm for Bayesian inference in this setting.By varying the simulated training data for the network,we can perform inference on a wide varietyof data types. This approach is therefore highlyflexible and improves on the state of the art interms of accuracy; provides uncertainty estimatesabout the particle locations and identities; and hasa test run-time that scales linearly as a functionof the data length and number of particles, thusenabling Bayesian inference in arbitrarily largeparticle tracking datasets.