HiST: A Hierarchical Sparse Transformer for Cross-Modal Spatial Transcriptomics Modeling

Spatial transcriptomics (ST) links gene expression with tissue morphology but remains expensive and low-throughput, motivating surrogates that infer expression from routine histology. Whole-slide H&E-to-ST inference pairs a gigapixel image with gene measurements at a sparse, irregular set of locations, making multiscale modeling challenging without incurring dense-grid overhead or quadratic token mixing. We propose HiST, a hierarchical sparse transformer that treats measured locations as a lattice-indexed sparse field and builds a dyadic encoder--decoder directly on the active tissue footprint. HiST combines sparse window attention for local geometric correspondence with resolution-changing operators for rapid multiscale context integration. For a fixed window size, the dominant runtime and memory scale with the number of observed locations rather than the dense slide area. To mitigate slide-specific acquisition variation, HiST adds a bottlenecked global conditioning pathway via a \emph{slide calibration token} that summarizes slide-level context and conditions local representations. On a multi-organ benchmark spanning diverse tissues and acquisition sources, HiST improves predictive performance over recent baselines while reducing runtime and peak memory.

Understanding how genes are switched on or off in different parts of a tissue helps scientists study diseases like cancer and figure out how organs work. The most informative way to measure this, called spatial transcriptomics, is slow and expensive, so most hospitals and labs cannot afford to run it on every tissue sample. In contrast, routine microscope images of tissue slices, the kind pathologists already look at every day, are cheap and widely available, but they do not directly reveal which genes are active. In this paper, we build a computer model that looks at a routine tissue image and predicts what the gene-activity measurements at each location would have shown, without anyone needing to run the expensive experiment. Compared to existing methods, our approach handles full microscope slides much faster and with less computer memory, while giving more accurate predictions. If reliable, tools like this could let researchers ask gene-activity questions at the scale of large hospital archives, accelerating biological discovery without requiring every sample to undergo costly molecular measurement.