Abstract
Background. Cross-platform transcriptomic harmonization benchmarks rely on individual success metric each: correlation with a reference profile, classifier accuracy, or a composite imported from single-cell integration. We found no published study that computes multiple orthogonal metric classes over one set of harmonization approaches and measures overlapping between the best approaches by these metrics.
Methods. We used the 2,234 approaches of the recent real-world benchmark on 7,174 B-cell lymphoma transcriptomes. They combine 31 harmonization methods, 14 batch-removal strategies, three imputations and two post-removal settings. We assigned its 87 polarity-adjusted scoring metrics, out of 344 registered, to five classes (global, local, distributional, structural and composite) and added three biology-facing classes: within-cohort marker correlation (L), cross-batch rank agreement of a marker panel (M), and leave-one-batch-out (LOBO) prediction of lymphoma subtypes and normal B cells (N). Each class resulted in top 112 approaches (5%), and we measured overlap by the Jaccard index.
Results. Harmonization methods of high harshness such as quantile normalization matched different batches to the same shape of distribution but were inferior by local and global metrics. Local and composite metrics resulted in the largest overlap of best approaches (71 shared). Classes L and M, M and N shared no approach, whereas L and N shared one approach without harmonization under the rare-batches-removed strategy, and all three overlaps were significantly below the random expectation. Class N resulted in the ComBat family in 84 of its 112 places. Class M elected only 10_mnn and 12_scanorama, methods that completely corrupted biology by correlation.
Conclusions. Best transcriptomic harmonization approach is highly contingent on the quality metrics used, with metric classes trade-offing between each other. We provide a six-step recipe for metric classes, their thresholds and the cross-batch tests for selecting the best approaches.