Abstract
Abstract This paper aims to design and evaluate a text-speech-image multimodal collaborative English translation system. Addressing the issues of coarse-grained modal semantic alignment and the difficulty in balancing inference efficiency and translation quality in current text-speech-image multimodal English translation systems, a multimodal collaborative translation system based on fine-grained cross-modal alignment and an adaptive gating fusion mechanism is proposed. First, Vision Transformer (ViT)-Base, wav2vec 2.0, and multilingual BART with 50 languages (mBART-50) are used to extract context-aware features of image regions, speech frames, and text words, respectively. Fine-grained alignment of triples is achieved through contrast loss in the shared semantic space and a cross-modal mask reconstruction task. Then, a modality-aware gating network is designed to dynamically adjust the fusion weights of image and speech at the word level, guided by text, and a single-layer lightweight cross-attention module with fixed structural parameters is embedded to reduce decoding overhead. A two-stage training strategy is employed. First, end-to-end fine-tuning is used to jointly optimize the negative log-likelihood, contrastive, and reconstruction losses. Then, a policy gradient-based REINFORCE algorithm with specific sampling and variance reduction strategies is applied for optimization, jointly using Bilingual Evaluation Understudy (BLEU), Contrastive Language-Image Pretraining Score (CLIPScore), and latency-aware reward. Experiments on the 3AM and Flickr8k-En datasets demonstrate that the proposed method outperforms baseline models across multiple metrics, including text quality, cross-modal consistency, speech adaptability, and system efficiency. Specifically, BLEU-4 scores reach 42.7; CLIPScore is 0.802; Word Error Rate (WER) is 8.3%; end-to-end latency is 420 ms. This method effectively improves the disambiguation capability, cross-modal consistency, and system real-time performance of multimodal translation, and preliminarily verifies the balanced advantage of the proposed architecture in multimodal semantic collaborative modeling and practical deployment scenarios.