Benchmarking single-cell foundation models reveals an advantage in the low-data regime for cell-type and disease-state prediction

Abstract Single-cell foundation models perform well on cell-type classification, but their value for disease-state prediction and their advantage over classical representations remains unclear. We benchmarked cell-representations from four foundation models using logistic regression, on 868 disease-prediction tasks (82 datasets) and 65 cell-type datasets from CELLxGENE. With full data, the baseline (lognormalized counts) outperformed the best model (TranscriptFormer-Sapiens) for cell type and matched it for disease. However, with few labeled cells, TranscriptFormer-Sapiens significantly outperformed the baseline in both tasks up to ∼ 10 2 samples per class. Our comprehensive zero-shot benchmark identifies the data-scarce regime as a domain where Single-cell foundation models truly show advantage.