Slide-free microscopy (SFM) methods can serve as a faster alternative to the standard histological examination of tissue specimens. However, SFM methods often provide images that differ from the hematoxylin- and eosin-stained (H\&E) images commonly obtained in standard histology. Unpaired image-to-image translation has been explored for transforming SFM images into H\&E images, a process known as virtual staining. Here, we compare a standard CycleGAN approach to a diffusion model-based approach for virtual staining of SFM images. We observe that the diffusion model approach, which relies on the inherent semantic preservation of the latent encodings, fails to outperform the standard CycleGAN approach, when tested on two different SFM datasets. This indicates that the semantic preservation of diffusion models is lacking for virtual staining tasks and additional regularization is needed.