Deep learning prediction of radiation-induced xerostomia and personalized outcome-driven treatment planning in head and neck cancer

Abstract Objective . Radiation therapy (RT) is critical in head and neck cancer (HNC) treatment but often causes radiation-induced toxicities, such as xerostomia (RIX). While deep learning (DL)-based models show promise in predicting these toxicities, their black-box nature hinders clinical applications. This study aims to develop a robust DL model to predict RIX 12 months after RT and to leverage model interpretability techniques, specifically class activation maps (CAMs) and voxel-wise dose gradient maps (GMs), to guide personalized treatment plan optimization. Approach . A 3D ResNet-based model was trained using planning CTs, dose volumes, and salivary gland contours obtained from a retrospective cohort of 839 HNC patients. To address anatomical variations, we normalized each patient’s volumes to a common reference frame using atlas normalization. To improve spatial correspondence between deep features and patient anatomy, we integrated blur pooling and adaptive average pooling, mitigating downsampling-induced voxel shifts. Model interpretability was achieved using Grad-CAM++ and GMs. Personalized plan optimization was performed on predicted RIX-positive cases by generating avoidance contours from both CAMs and GMs to guide dose reduction. Main results . The proposed model was tested on 30 independent test cases. It achieved an AUC of 0.77 with balanced sensitivity (0.71) and specificity (0.83). Among 9 predicted RIX-positive cases, GM-guided optimization converted the predictions to RIX-negative in 7 cases (77.8%), compared with 6 cases (66.7%) for CAM-guided optimization, and achieved a greater reduction in mean model-predicted RIX probability (26% versus 20%). Significance . The proposed atlas-normalized model achieved robust discrimination, and a 9-case plan-refinement analysis showed that CAM- and GM-derived spatial information could be translated into clinically constrained avoidance objectives that reduced the model-predicted RIX probability while preserving specified target and organs at risk dose requirements. These results demonstrate the feasibility of the proposed strategy for guiding clinical interventions.