Abstract: Stochastic natural gradient variational inference (SNGVI) is a popular method for posterior inference on large datasets with many applications. Despite its wide usage over the years, little is known about its non-asymptotic convergence rate in the *stochastic* setting. We aim to close this gap by providing a better understanding of the convergence of stochastic natural gradient variational inference. For conjugate likelihoods, we prove the first $\mathcal{O}(\frac{1}{T})$ non-asymptotic convergence rate of SNGVI, which is no slower than stochastic gradient descent (a.k.a. BBVI), and has better constant dependency that leads to faster convergence in practice. For non-conjugate likelihoods, we show that (canonical) NGVI implicitly optimizes a non-convex objective. Therefore, a global convergence rate of $\mathcal{O}(\frac{1}{T})$ is unlikely without some significant new understanding of optimizing the ELBO using natural gradients.