Thank you all for taking the time to thoroughly read and comment on this paper. Your comments and criticisms are very much appreciated. $
There are a few important points to be made in response.

1. The known mixing rates for the random scan Gibbs sampler on RBMs do not necessarily encompass the ones provided in Section 3 for the alternating Gibbs sampler. Because it was explicitly mentioned, let us contrast the results of Section 3 with Liu and Domke (2014) “Projecting MRFs for fast mixing.”

In the context of an RBM with weight matrix W, Liu and Domke guarantee there if there exists a matrix norm | | for which |W| < 1, then the random scan Gibbs sampler mixes in time order of 1/(1-|W|). Note that for a matrix W, |W|_1 * |W^T|_1 is not a matrix norm and thus the results from that paper do not hold with respect to this quantity. On the other hand, Section 3 shows that if |W|_1 * |W^T|_1 < 4, then the alternating Gibbs sampler mixes in time order of 1/(log(4) - log(|W|_1 * |W^T|_1)).

However, there are matrices W for which 1/(log(4) - log(|W|_1 * |W^T|_1)) << 1/(1-|W|) for any matrix norm | |. For these cases, Section 3 guarantees faster mixing for the alternating Gibbs sampler than can be guaranteed for the random scan Gibbs sampler.

Because contrastive divergence is run with relatively few iterations of the Gibbs sampler, this may help explain why the alternating Gibbs sampler is more effective than the random Gibbs sampler in practice.


2. The results in Section 4 for Gaussian-NReLU RBMs appear to be the first explicit bounds for Gibbs sampler over this variant. This is important since Gaussian-NReLU RBMs appear to be useful in practice, but theoretical guarantees have been lacking.


3. The results in Sections 4 and 5 give an almost complete worst-case characterization for the conditions necessary to guarantee rapid mixing for the alternating Gibbs sampler over Gaussian-Gaussian RBMs. This result appears to be the first of its kind for any variant of RBMs and helps to explain why Gibbs sampling is ineffective for high-dimensional Gaussian-Gaussian RBMs in practice.

Thank you again for your time.