Abstract
Recent foundation models for single-cell transcriptomic data generate informative, context-aware gene and cell representations. Spatial transcriptomic (ST) data offer extra positional insights but were not considered by these single-cell models. We introduce stFormer, a transformer model tailored for ST data, which employs the cross-attention module to incorporate spatial ligand genes. To unify different ST technologies with a trade-off between resolution and gene coverage, we propose a biased cross-attention method that enables the model to do learning with single-cell resolution on whole-transcriptome but low-resolution Visium data. We assembled a pretraining corpus comprising ∼4.1 million spatial samples from public human Visium datasets. After pretraining, stFormer improved upon the state-of-the-art single-cell foundation model, scFoundation, across diverse tasks, including cell clustering, batch effect correction, cell type prediction, and gene function prediction. stFormer also revealed intercellular ligand-receptor signaling responses via in silico perturbation. Finally, we demonstrated stFormer's utility in yielding biological findings on two case studies.