Abstract
Abstract In remote sensing image detection, small target features are easily lost during downsampling, and complex backgrounds can interfere with global semantic extraction, and target directions are arbitrarily arranged, further making rotation positioning difficult. To this end, the study proposes an improved You Only Look Once version 11 (YOLOv11) backbone and a Real-Time Detection Transformer (RT-DETR) for target detection. First, a space-depth conversion convolution reconstruction feature extraction network is introduced for YOLOv11 to losslessly map the spatial dimension information to the channel dimension. Secondly, an efficient hybrid encoder integrating RT-DETR is used to capture long-distance global dependencies using the AIFI and CCGM. Subsequently, a decoupled rotation detection head and a Gaussian distribution angle regression strategy are used to jointly optimize rotating target detection. Finally, systematic module ablation is compared and verified with mainstream algorithms on the DOTA and HRSC2016 benchmark data sets. On the two datasets, the mean accuracy was 81.68% and 94.45%, respectively, representing improvements of 5.26% and 5.95% over the baseline model. Furthermore, the model had 38.2 M parameters and an inference speed of 62 FPS, demonstrating a good balance between detection accuracy and computational efficiency. In the fine-grained performance analysis, the accuracy in difficult categories such as “small-vehicle” and “bridge” reached 80.5% and 58.2%, respectively, which was significantly better than that of the comparison algorithm. Compared with the Oriented Region-based Convolutional Neural Network, its precision on the DOTA dataset was 1.23% higher, and its inference speed was 2.8 times higher. Under the strict positioning standard of IoU = 0.8, its detection accuracy was 10.2% higher than that of the YOLOv8-OBB. The proposed model not only performs well in multi-scale feature extraction and global context awareness, but can also precisely position rotating targets.