Abstract
This paper presents a general scheme for enhancing the convergence and performance of DETR (DEtection TRansformer).We investigate the slow convergence problem in transformers from a new perspective, suggesting that it arises from the self-attention that introduces no structural bias on the inputs. To address this issue, we explore incorporating position relation prior as attention bias to augment object detection, following the verification of its statistical significance using a proposed quantitative macroscopic correlation (MC) metric. Our approach introduces an encoder to construct position relation embeddings for progressive attention refinement, which further extends the traditional streaming pipeline of DETR into a contrastive relation pipeline to address the conflicts between non-duplicate predictions and positive supervision. During the progressive refinement process, pattern collapse in multi-layer relations is investigated and alleviated through explicit layer-wise encoding and gated relation modulation. The proposed approach, termed Relation-DETR+, establishes a streamlined relation-enhanced pipeline for dense prediction, capable of jointly addressing object detection, semantic segmentation, instance segmentation, and panoptic segmentation tasks within a single framework. Extensive experiments on both generic and task-specific datasets demonstrate the effectiveness of our approach. Under the same training schedules, our approach demonstrates superior learning efficiency compared to DINO in detection and Mask-DINO in segmentation, respectively. Moreover, the proposed relation encoder serves as a universal plug-and-play component, bringing clear improvements for theoretically any DETR-like methods. Furthermore, we introduce a large-scale class-agnostic detection dataset, SA-Det-100 k. The experimental results on the dataset illustrate the potential of proposed explicit position relation towards universal object detection.