Abstract
Abstract Automatic understanding of multi-instrument music requires the joint interpretation of score images, harmonic relations, and audio-side instrument identities, yet these tasks are commonly optimized in isolation. This study proposes a modular, interface-coupled framework for score transcription, harmony-aware symbolic reconstruction, and multi-instrument recognition. The framework consists of a multi-scale residual neural network (MSR-NN), a temporal harmonic graph convolutional network (THGCN), and a dual-branch CNN-DCNN. MSR-NN preserves staff-position and note-contour information through bottom-up residual extraction and top-down shallow–deep feature fusion. THGCN constructs graph edges according to integer harmonic ratios in the log-frequency domain and uses a GRU to model temporal continuity. CNN-DCNN combines standard and dilated convolutions to capture local timbral cues and broader spectro-temporal context without reducing Mel-spectrogram resolution. The three outputs are coupled through a constrained symbolic decoder rather than a fully joint end-to-end network. Under the PrIMuS and GrandStaff protocols, MSR-NN reduces the symbol error rate to 3.82%, representing an improvement of 0.40–4.09 percentage points over the selected baselines. Component ablation increases SER from 3.82% to 4.37% after removing multi-scale fusion and to 4.68% after removing residual learning. Replacing THGCN with temporal-only modeling increases F0-RMSE from 3.52 to 4.07, while replacing CNN-DCNN with a standard CNN reduces macro-F1 from 0.742 to 0.712. Symbolic refinement improves sequence consistency from 88.3% to 94.3%. On IRMAS, CNN-DCNN obtains an accuracy of 0.929, a macro-F1 of 0.742, and a macro-AUC of 0.934. External and perturbation tests obtain an AUC of 0.846 on the common-label OpenMIC-2018 transfer setting and 0.911 under 10-dB additive noise. Five-seed paired tests remain significant after Holm correction (adjusted p < 0.05). These results demonstrate that the contribution lies in music-specific cross-module coupling and constrained symbolic reconstruction rather than in introducing residual, graph, recurrent, or dilated convolution as isolated operators.